{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distributed data parallel MNIST training with TensorFlow2 and SMDataParallel\n",
    "\n",
    "SMDataParallel is a new capability in Amazon SageMaker to train deep learning models faster and cheaper. SMDataParallel is a distributed data parallel training framework for TensorFlow2, PyTorch and MXNet.  \n",
    "\n",
    "This notebook example shows how to use SMDataParallel with TensorFlow2 in SageMaker using MNIST dataset.\n",
    "\n",
    "For more information:\n",
    "1. [TensorFlow in SageMaker](https://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/using_tf.html)\n",
    "2. [SMDataParallelTensorFlow API Specification] < LINK TO BE ADDED >\n",
    "3. [Getting started with SMDataParallel on SageMaker] < LINK TO BE ADDED >\n",
    "\n",
    "**NOTE:** This example requires SageMaker Python SDK v2.X.\n",
    "\n",
    "### Dataset\n",
    "This example uses the MNIST dataset. MNIST is a widely used dataset for handwritten digit classification. It consists of 70,000 labeled 28x28 pixel grayscale images of handwritten digits. The dataset is split into 60,000 training images and 10,000 test images. There are 10 classes (one for each of the 10 digits).\n",
    "\n",
    "### SageMaker execution roles\n",
    "\n",
    "The IAM role arn used to give training and hosting access to your data. See the [Amazon SageMaker Roles](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html) for how to create these. Note, if more than one role is required for notebook instances, training, and/or hosting, please replace the sagemaker.get_execution_role() with the appropriate full IAM role arn string(s).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: sagemaker==2.19.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (2.19.0)\n",
      "Requirement already satisfied: boto3>=1.16.32 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from sagemaker==2.19.0) (1.16.33)\n",
      "Requirement already satisfied: attrs in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from sagemaker==2.19.0) (19.3.0)\n",
      "Requirement already satisfied: protobuf>=3.1 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from sagemaker==2.19.0) (3.8.0)\n",
      "Requirement already satisfied: google-pasta in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from sagemaker==2.19.0) (0.2.0)\n",
      "Requirement already satisfied: smdebug-rulesconfig>=1.0.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from sagemaker==2.19.0) (1.0.0)\n",
      "Requirement already satisfied: importlib-metadata>=1.4.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from sagemaker==2.19.0) (1.5.0)\n",
      "Requirement already satisfied: packaging>=20.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from sagemaker==2.19.0) (20.1)\n",
      "Requirement already satisfied: numpy>=1.9.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from sagemaker==2.19.0) (1.18.1)\n",
      "Requirement already satisfied: protobuf3-to-dict>=0.1.5 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from sagemaker==2.19.0) (0.1.5)\n",
      "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from boto3>=1.16.32->sagemaker==2.19.0) (0.3.3)\n",
      "Requirement already satisfied: botocore<1.20.0,>=1.19.33 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from boto3>=1.16.32->sagemaker==2.19.0) (1.19.33)\n",
      "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from boto3>=1.16.32->sagemaker==2.19.0) (0.10.0)\n",
      "Requirement already satisfied: six>=1.9 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from protobuf>=3.1->sagemaker==2.19.0) (1.14.0)\n",
      "Requirement already satisfied: setuptools in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from protobuf>=3.1->sagemaker==2.19.0) (45.2.0.post20200210)\n",
      "Requirement already satisfied: zipp>=0.5 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from importlib-metadata>=1.4.0->sagemaker==2.19.0) (2.2.0)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from packaging>=20.0->sagemaker==2.19.0) (2.4.6)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.25.4; python_version != \"3.4\" in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from botocore<1.20.0,>=1.19.33->boto3>=1.16.32->sagemaker==2.19.0) (1.25.10)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from botocore<1.20.0,>=1.19.33->boto3>=1.16.32->sagemaker==2.19.0) (2.8.1)\n",
      "\u001b[33mWARNING: You are using pip version 20.0.2; however, version 20.3.1 is available.\n",
      "You should consider upgrading via the '/home/ec2-user/anaconda3/envs/tensorflow_p36/bin/python -m pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install sagemaker==2.19.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "\n",
    "sagemaker_session = sagemaker.Session()\n",
    "role = sagemaker.get_execution_role()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model training with SMDataParallel\n",
    "\n",
    "### Training script\n",
    "\n",
    "The `train_tensorflow_smdataparallel_mnist.py` script provides the code you need for training a SageMaker model using SMDataParallel's `DistributedGradientTape`. The training script is very similar to a TensorFlow2 training script you might run outside of SageMaker, but modified to run with SMDataParallel. SMDataParallel's TensorFlow client provides an alternative to native `DistributedGradientTape`. For details about how to use SMDataParallel in your native TF2 script, see the Getting started with SMDataParallel tutorial.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[37m# Licensed to the Apache Software Foundation (ASF) under one\u001b[39;49;00m\r\n",
      "\u001b[37m# or more contributor license agreements.  See the NOTICE file\u001b[39;49;00m\r\n",
      "\u001b[37m# distributed with this work for additional information\u001b[39;49;00m\r\n",
      "\u001b[37m# regarding copyright ownership.  The ASF licenses this file\u001b[39;49;00m\r\n",
      "\u001b[37m# to you under the Apache License, Version 2.0 (the\u001b[39;49;00m\r\n",
      "\u001b[37m# \"License\"); you may not use this file except in compliance\u001b[39;49;00m\r\n",
      "\u001b[37m# with the License.  You may obtain a copy of the License at\u001b[39;49;00m\r\n",
      "\u001b[37m#\u001b[39;49;00m\r\n",
      "\u001b[37m#   http://www.apache.org/licenses/LICENSE-2.0\u001b[39;49;00m\r\n",
      "\u001b[37m#\u001b[39;49;00m\r\n",
      "\u001b[37m# Unless required by applicable law or agreed to in writing,\u001b[39;49;00m\r\n",
      "\u001b[37m# software distributed under the License is distributed on an\u001b[39;49;00m\r\n",
      "\u001b[37m# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\u001b[39;49;00m\r\n",
      "\u001b[37m# KIND, either express or implied.  See the License for the\u001b[39;49;00m\r\n",
      "\u001b[37m# specific language governing permissions and limitations\u001b[39;49;00m\r\n",
      "\u001b[37m# under the License.\u001b[39;49;00m\r\n",
      "\r\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mtensorflow\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m \u001b[04m\u001b[36mtf\u001b[39;49;00m\r\n",
      "\r\n",
      "\u001b[37m# Import SMDataParallel TensorFlow2 Modules\u001b[39;49;00m\r\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36msmdistributed\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mdataparallel\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mtensorflow\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m \u001b[04m\u001b[36mdist\u001b[39;49;00m\r\n",
      "\r\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mos\u001b[39;49;00m\r\n",
      "\r\n",
      "tf.random.set_seed(\u001b[34m42\u001b[39;49;00m)\r\n",
      "\r\n",
      "\r\n",
      "\u001b[37m# SMDataParallel: Initialize\u001b[39;49;00m\r\n",
      "dist.init()\r\n",
      "\r\n",
      "gpus = tf.config.experimental.list_physical_devices(\u001b[33m'\u001b[39;49;00m\u001b[33mGPU\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\r\n",
      "\u001b[34mfor\u001b[39;49;00m gpu \u001b[35min\u001b[39;49;00m gpus:\r\n",
      "    tf.config.experimental.set_memory_growth(gpu, \u001b[34mTrue\u001b[39;49;00m)\r\n",
      "\u001b[34mif\u001b[39;49;00m gpus:\r\n",
      "    \u001b[37m# SMDataParallel: Pin GPUs to a single SMDataParallel process [use SMDataParallel local_rank() API]\u001b[39;49;00m\r\n",
      "    tf.config.experimental.set_visible_devices(gpus[dist.local_rank()], \u001b[33m'\u001b[39;49;00m\u001b[33mGPU\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\r\n",
      "\r\n",
      "(mnist_images, mnist_labels), _ = \\\r\n",
      "    tf.keras.datasets.mnist.load_data(path=\u001b[33m'\u001b[39;49;00m\u001b[33mmnist-\u001b[39;49;00m\u001b[33m%d\u001b[39;49;00m\u001b[33m.npz\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m % dist.rank())\r\n",
      "\r\n",
      "dataset = tf.data.Dataset.from_tensor_slices(\r\n",
      "    (tf.cast(mnist_images[..., tf.newaxis] / \u001b[34m255.0\u001b[39;49;00m, tf.float32),\r\n",
      "             tf.cast(mnist_labels, tf.int64))\r\n",
      ")\r\n",
      "dataset = dataset.repeat().shuffle(\u001b[34m10000\u001b[39;49;00m).batch(\u001b[34m128\u001b[39;49;00m)\r\n",
      "\r\n",
      "mnist_model = tf.keras.Sequential([\r\n",
      "    tf.keras.layers.Conv2D(\u001b[34m32\u001b[39;49;00m, [\u001b[34m3\u001b[39;49;00m, \u001b[34m3\u001b[39;49;00m], activation=\u001b[33m'\u001b[39;49;00m\u001b[33mrelu\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m),\r\n",
      "    tf.keras.layers.Conv2D(\u001b[34m64\u001b[39;49;00m, [\u001b[34m3\u001b[39;49;00m, \u001b[34m3\u001b[39;49;00m], activation=\u001b[33m'\u001b[39;49;00m\u001b[33mrelu\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m),\r\n",
      "    tf.keras.layers.MaxPooling2D(pool_size=(\u001b[34m2\u001b[39;49;00m, \u001b[34m2\u001b[39;49;00m)),\r\n",
      "    tf.keras.layers.Dropout(\u001b[34m0.25\u001b[39;49;00m),\r\n",
      "    tf.keras.layers.Flatten(),\r\n",
      "    tf.keras.layers.Dense(\u001b[34m128\u001b[39;49;00m, activation=\u001b[33m'\u001b[39;49;00m\u001b[33mrelu\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m),\r\n",
      "    tf.keras.layers.Dropout(\u001b[34m0.5\u001b[39;49;00m),\r\n",
      "    tf.keras.layers.Dense(\u001b[34m10\u001b[39;49;00m, activation=\u001b[33m'\u001b[39;49;00m\u001b[33msoftmax\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\r\n",
      "])\r\n",
      "loss = tf.losses.SparseCategoricalCrossentropy()\r\n",
      "\r\n",
      "\u001b[37m# SMDataParallel: dist.size()\u001b[39;49;00m\r\n",
      "\u001b[37m# LR for 8 node run : 0.000125\u001b[39;49;00m\r\n",
      "\u001b[37m# LR for single node run : 0.001\u001b[39;49;00m\r\n",
      "opt = tf.optimizers.Adam(\u001b[34m0.000125\u001b[39;49;00m * dist.size())\r\n",
      "\r\n",
      "checkpoint_dir = os.environ[\u001b[33m'\u001b[39;49;00m\u001b[33mSM_MODEL_DIR\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m]\r\n",
      "\r\n",
      "checkpoint = tf.train.Checkpoint(model=mnist_model, optimizer=opt)\r\n",
      "\r\n",
      "\r\n",
      "\u001b[90m@tf\u001b[39;49;00m.function\r\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32mtraining_step\u001b[39;49;00m(images, labels, first_batch):\r\n",
      "    \u001b[34mwith\u001b[39;49;00m tf.GradientTape() \u001b[34mas\u001b[39;49;00m tape:\r\n",
      "        probs = mnist_model(images, training=\u001b[34mTrue\u001b[39;49;00m)\r\n",
      "        loss_value = loss(labels, probs)\r\n",
      "\r\n",
      "    \u001b[37m# SMDataParallel: Wrap tf.GradientTape with SMDataParallel's DistributedGradientTape\u001b[39;49;00m\r\n",
      "    tape = dist.DistributedGradientTape(tape)\r\n",
      "\r\n",
      "    grads = tape.gradient(loss_value, mnist_model.trainable_variables)\r\n",
      "    opt.apply_gradients(\u001b[36mzip\u001b[39;49;00m(grads, mnist_model.trainable_variables))\r\n",
      "\r\n",
      "    \u001b[34mif\u001b[39;49;00m first_batch:\r\n",
      "       \u001b[37m# SMDataParallel: Broadcast model and optimizer variables\u001b[39;49;00m\r\n",
      "       dist.broadcast_variables(mnist_model.variables, root_rank=\u001b[34m0\u001b[39;49;00m)\r\n",
      "       dist.broadcast_variables(opt.variables(), root_rank=\u001b[34m0\u001b[39;49;00m)\r\n",
      "    \r\n",
      "    \u001b[37m# SMDataParallel: all_reduce call\u001b[39;49;00m\r\n",
      "    loss_value = dist.oob_allreduce(loss_value) \u001b[37m# Average the loss across workers\u001b[39;49;00m\r\n",
      "    \u001b[34mreturn\u001b[39;49;00m loss_value\r\n",
      "\r\n",
      "\r\n",
      "\u001b[34mfor\u001b[39;49;00m batch, (images, labels) \u001b[35min\u001b[39;49;00m \u001b[36menumerate\u001b[39;49;00m(dataset.take(\u001b[34m10000\u001b[39;49;00m // dist.size())):\r\n",
      "    loss_value = training_step(images, labels, batch == \u001b[34m0\u001b[39;49;00m)\r\n",
      "\r\n",
      "    \u001b[34mif\u001b[39;49;00m batch % \u001b[34m50\u001b[39;49;00m == \u001b[34m0\u001b[39;49;00m \u001b[35mand\u001b[39;49;00m dist.rank() == \u001b[34m0\u001b[39;49;00m:\r\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mStep #\u001b[39;49;00m\u001b[33m%d\u001b[39;49;00m\u001b[33m\\t\u001b[39;49;00m\u001b[33mLoss: \u001b[39;49;00m\u001b[33m%.6f\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m % (batch, loss_value))\r\n",
      "\r\n",
      "\u001b[37m# SMDataParallel: Save checkpoints only from master node.\u001b[39;49;00m\r\n",
      "\u001b[34mif\u001b[39;49;00m dist.rank() == \u001b[34m0\u001b[39;49;00m:\r\n",
      "    mnist_model.save(os.path.join(checkpoint_dir, \u001b[33m'\u001b[39;49;00m\u001b[33m1\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m))\r\n"
     ]
    }
   ],
   "source": [
    "!pygmentize code/train_tensorflow_smdataparallel_mnist.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SageMaker TensorFlow Estimator function options\n",
    "\n",
    "\n",
    "In the following code block, you can update the estimator function to use a different instance type, instance count, and distrubtion strategy. You're also passing in the training script you reviewed in the previous cell.\n",
    "\n",
    "**Instance types**\n",
    "\n",
    "SMDataParallel supports model training on SageMaker with the following instance types only:\n",
    "1. ml.p3.16xlarge\n",
    "1. ml.p3dn.24xlarge [Recommended]\n",
    "1. ml.p4d.24xlarge [Recommended]\n",
    "\n",
    "**Instance count**\n",
    "\n",
    "To get the best performance and the most out of SMDataParallel, you should use at least 2 instances, but you can also use 1 for testing this example.\n",
    "\n",
    "**Distribution strategy**\n",
    "\n",
    "Note that to use DDP mode, you update the the `distribution` strategy, and set it to use `smdistributed dataparallel`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tensorflow import TensorFlow\n",
    "estimator = TensorFlow(\n",
    "                        base_job_name='tensorflow2-smdataparallel-mnist',\n",
    "                        source_dir='code',\n",
    "                        entry_point='train_tensorflow_smdataparallel_mnist.py',\n",
    "                        role=role,\n",
    "                        py_version='py37',\n",
    "                        framework_version='2.3.1',\n",
    "                        # For training with multinode distributed training, set this count. Example: 2\n",
    "                        instance_count=2,\n",
    "                        # For training with p3dn instance use - ml.p3dn.24xlarge\n",
    "                        instance_type= 'ml.p3.16xlarge',\n",
    "                        sagemaker_session=sagemaker_session,\n",
    "                        # Training using SMDataParallel Distributed Training Framework\n",
    "                        distribution={'smdistributed':{\n",
    "                                            'dataparallel':{\n",
    "                                                    'enabled': True\n",
    "                                             }\n",
    "                                      }}\n",
    "                        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-12-10 20:14:09 Starting - Starting the training job...\n",
      "2020-12-10 20:14:32 Starting - Launching requested ML instancesProfilerReport-1607631248: InProgress\n",
      ".........\n",
      "2020-12-10 20:15:57 Starting - Preparing the instances for training.........\n",
      "2020-12-10 20:17:38 Downloading - Downloading input data...\n",
      "2020-12-10 20:17:54 Training - Downloading the training image...........\u001b[35m2020-12-10 20:19:46.916346: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:460] Initializing the SageMaker Profiler.\u001b[0m\n",
      "\u001b[35m2020-12-10 20:19:46.927920: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:105] SageMaker Profiler is not enabled. The timeline writer thread will not be started, future recorded events will be dropped.\u001b[0m\n",
      "\u001b[35m2020-12-10 20:19:47.302436: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0\u001b[0m\n",
      "\u001b[35m2020-12-10 20:19:47.429051: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:460] Initializing the SageMaker Profiler.\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:45.816654: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:460] Initializing the SageMaker Profiler.\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:45.824053: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:105] SageMaker Profiler is not enabled. The timeline writer thread will not be started, future recorded events will be dropped.\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:46.161140: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:46.277158: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:460] Initializing the SageMaker Profiler.\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:50,095 sagemaker-training-toolkit INFO     Imported framework sagemaker_tensorflow_container.training\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:50,847 sagemaker-training-toolkit INFO     Starting MPI run as worker node.\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:50,847 sagemaker-training-toolkit INFO     Creating SSH daemon.\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:50,854 sagemaker-training-toolkit INFO     Waiting for MPI workers to establish their SSH connections\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:50,856 sagemaker-training-toolkit INFO     Cannot connect to host algo-2 at port 22\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:50,857 sagemaker-training-toolkit ERROR    Connection failed\u001b[0m\n",
      "\u001b[34mTraceback (most recent call last):\n",
      "  File \"/usr/local/lib/python3.7/site-packages/sagemaker_training/smdataparallel.py\", line 303, in _can_connect\n",
      "    client.connect(host, port=port)\n",
      "  File \"/usr/local/lib/python3.7/site-packages/paramiko/client.py\", line 368, in connect\n",
      "    raise NoValidConnectionsError(errors)\u001b[0m\n",
      "\u001b[34mparamiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 10.2.228.56\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:50,859 sagemaker-training-toolkit INFO     Connection closed\u001b[0m\n",
      "\u001b[35m2020-12-10 20:19:51,315 sagemaker-training-toolkit INFO     Imported framework sagemaker_tensorflow_container.training\u001b[0m\n",
      "\u001b[35m2020-12-10 20:19:52,042 sagemaker-training-toolkit INFO     Starting MPI run as worker node.\u001b[0m\n",
      "\u001b[35m2020-12-10 20:19:52,042 sagemaker-training-toolkit INFO     Waiting for MPI Master to create SSH daemon.\u001b[0m\n",
      "\u001b[35m2020-12-10 20:19:52,051 paramiko.transport INFO     Connected (version 2.0, client OpenSSH_7.6p1)\u001b[0m\n",
      "\u001b[35m2020-12-10 20:19:52,124 paramiko.transport INFO     Authentication (publickey) successful!\u001b[0m\n",
      "\u001b[35m2020-12-10 20:19:52,124 sagemaker-training-toolkit INFO     Can connect to host algo-1\u001b[0m\n",
      "\u001b[35m2020-12-10 20:19:52,124 sagemaker-training-toolkit INFO     MPI Master online, creating SSH daemon.\u001b[0m\n",
      "\u001b[35m2020-12-10 20:19:52,124 sagemaker-training-toolkit INFO     Writing environment variables to /etc/environment for the MPI process.\u001b[0m\n",
      "\u001b[35m2020-12-10 20:19:52,134 sagemaker-training-toolkit INFO     Waiting for MPI process to finish.\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:51,861 sagemaker-training-toolkit INFO     Cannot connect to host algo-2 at port 22\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:51,861 sagemaker-training-toolkit ERROR    Connection failed\u001b[0m\n",
      "\u001b[34mTraceback (most recent call last):\n",
      "  File \"/usr/local/lib/python3.7/site-packages/sagemaker_training/smdataparallel.py\", line 303, in _can_connect\n",
      "    client.connect(host, port=port)\n",
      "  File \"/usr/local/lib/python3.7/site-packages/paramiko/client.py\", line 368, in connect\n",
      "    raise NoValidConnectionsError(errors)\u001b[0m\n",
      "\u001b[34mparamiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 10.2.228.56\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:51,861 sagemaker-training-toolkit INFO     Connection closed\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:52,869 paramiko.transport INFO     Connected (version 2.0, client OpenSSH_7.6p1)\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:52,943 paramiko.transport INFO     Authentication (publickey) successful!\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:52,943 sagemaker-training-toolkit INFO     Can connect to host algo-2 at port 22\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:52,943 sagemaker-training-toolkit INFO     Connection closed\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:52,943 sagemaker-training-toolkit INFO     Worker algo-2 available for communication\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:52,943 sagemaker-training-toolkit INFO     Network interface name: eth0\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:52,943 sagemaker-training-toolkit INFO     Host: ['algo-1', 'algo-2']\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:52,943 sagemaker-training-toolkit INFO     instance type: ml.p3.16xlarge\u001b[0m\n",
      "\u001b[34m2020-12-10 20:19:53,032 sagemaker-training-toolkit INFO     Invoking user script\n",
      "\u001b[0m\n",
      "\u001b[34mTraining Env:\n",
      "\u001b[0m\n",
      "\u001b[34m{\n",
      "    \"additional_framework_parameters\": {\n",
      "        \"sagemaker_distributed_dataparallel_enabled\": true,\n",
      "        \"sagemaker_instance_type\": \"ml.p3.16xlarge\"\n",
      "    },\n",
      "    \"channel_input_dirs\": {},\n",
      "    \"current_host\": \"algo-1\",\n",
      "    \"framework_module\": \"sagemaker_tensorflow_container.training:main\",\n",
      "    \"hosts\": [\n",
      "        \"algo-1\",\n",
      "        \"algo-2\"\n",
      "    ],\n",
      "    \"hyperparameters\": {\n",
      "        \"model_dir\": \"s3://sagemaker-us-east-1-835319576252/tensorflow2-smdataparallel-mnist-2020-12-10-20-14-08-681/model\"\n",
      "    },\n",
      "    \"input_config_dir\": \"/opt/ml/input/config\",\n",
      "    \"input_data_config\": {},\n",
      "    \"input_dir\": \"/opt/ml/input\",\n",
      "    \"is_master\": true,\n",
      "    \"job_name\": \"tensorflow2-smdataparallel-mnist-2020-12-10-20-14-08-681\",\n",
      "    \"log_level\": 20,\n",
      "    \"master_hostname\": \"algo-1\",\n",
      "    \"model_dir\": \"/opt/ml/model\",\n",
      "    \"module_dir\": \"s3://sagemaker-us-east-1-835319576252/tensorflow2-smdataparallel-mnist-2020-12-10-20-14-08-681/source/sourcedir.tar.gz\",\n",
      "    \"module_name\": \"train_tensorflow_smdataparallel_mnist\",\n",
      "    \"network_interface_name\": \"eth0\",\n",
      "    \"num_cpus\": 64,\n",
      "    \"num_gpus\": 8,\n",
      "    \"output_data_dir\": \"/opt/ml/output/data\",\n",
      "    \"output_dir\": \"/opt/ml/output\",\n",
      "    \"output_intermediate_dir\": \"/opt/ml/output/intermediate\",\n",
      "    \"resource_config\": {\n",
      "        \"current_host\": \"algo-1\",\n",
      "        \"hosts\": [\n",
      "            \"algo-1\",\n",
      "            \"algo-2\"\n",
      "        ],\n",
      "        \"network_interface_name\": \"eth0\"\n",
      "    },\n",
      "    \"user_entry_point\": \"train_tensorflow_smdataparallel_mnist.py\"\u001b[0m\n",
      "\u001b[34m}\n",
      "\u001b[0m\n",
      "\u001b[34mEnvironment variables:\n",
      "\u001b[0m\n",
      "\u001b[34mSM_HOSTS=[\"algo-1\",\"algo-2\"]\u001b[0m\n",
      "\u001b[34mSM_NETWORK_INTERFACE_NAME=eth0\u001b[0m\n",
      "\u001b[34mSM_HPS={\"model_dir\":\"s3://sagemaker-us-east-1-835319576252/tensorflow2-smdataparallel-mnist-2020-12-10-20-14-08-681/model\"}\u001b[0m\n",
      "\u001b[34mSM_USER_ENTRY_POINT=train_tensorflow_smdataparallel_mnist.py\u001b[0m\n",
      "\u001b[34mSM_FRAMEWORK_PARAMS={\"sagemaker_distributed_dataparallel_enabled\":true,\"sagemaker_instance_type\":\"ml.p3.16xlarge\"}\u001b[0m\n",
      "\u001b[34mSM_RESOURCE_CONFIG={\"current_host\":\"algo-1\",\"hosts\":[\"algo-1\",\"algo-2\"],\"network_interface_name\":\"eth0\"}\u001b[0m\n",
      "\u001b[34mSM_INPUT_DATA_CONFIG={}\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_DATA_DIR=/opt/ml/output/data\u001b[0m\n",
      "\u001b[34mSM_CHANNELS=[]\u001b[0m\n",
      "\u001b[34mSM_CURRENT_HOST=algo-1\u001b[0m\n",
      "\u001b[34mSM_MODULE_NAME=train_tensorflow_smdataparallel_mnist\u001b[0m\n",
      "\u001b[34mSM_LOG_LEVEL=20\u001b[0m\n",
      "\u001b[34mSM_FRAMEWORK_MODULE=sagemaker_tensorflow_container.training:main\u001b[0m\n",
      "\u001b[34mSM_INPUT_DIR=/opt/ml/input\u001b[0m\n",
      "\u001b[34mSM_INPUT_CONFIG_DIR=/opt/ml/input/config\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_DIR=/opt/ml/output\u001b[0m\n",
      "\u001b[34mSM_NUM_CPUS=64\u001b[0m\n",
      "\u001b[34mSM_NUM_GPUS=8\u001b[0m\n",
      "\u001b[34mSM_MODEL_DIR=/opt/ml/model\u001b[0m\n",
      "\u001b[34mSM_MODULE_DIR=s3://sagemaker-us-east-1-835319576252/tensorflow2-smdataparallel-mnist-2020-12-10-20-14-08-681/source/sourcedir.tar.gz\u001b[0m\n",
      "\u001b[34mSM_TRAINING_ENV={\"additional_framework_parameters\":{\"sagemaker_distributed_dataparallel_enabled\":true,\"sagemaker_instance_type\":\"ml.p3.16xlarge\"},\"channel_input_dirs\":{},\"current_host\":\"algo-1\",\"framework_module\":\"sagemaker_tensorflow_container.training:main\",\"hosts\":[\"algo-1\",\"algo-2\"],\"hyperparameters\":{\"model_dir\":\"s3://sagemaker-us-east-1-835319576252/tensorflow2-smdataparallel-mnist-2020-12-10-20-14-08-681/model\"},\"input_config_dir\":\"/opt/ml/input/config\",\"input_data_config\":{},\"input_dir\":\"/opt/ml/input\",\"is_master\":true,\"job_name\":\"tensorflow2-smdataparallel-mnist-2020-12-10-20-14-08-681\",\"log_level\":20,\"master_hostname\":\"algo-1\",\"model_dir\":\"/opt/ml/model\",\"module_dir\":\"s3://sagemaker-us-east-1-835319576252/tensorflow2-smdataparallel-mnist-2020-12-10-20-14-08-681/source/sourcedir.tar.gz\",\"module_name\":\"train_tensorflow_smdataparallel_mnist\",\"network_interface_name\":\"eth0\",\"num_cpus\":64,\"num_gpus\":8,\"output_data_dir\":\"/opt/ml/output/data\",\"output_dir\":\"/opt/ml/output\",\"output_intermediate_dir\":\"/opt/ml/output/intermediate\",\"resource_config\":{\"current_host\":\"algo-1\",\"hosts\":[\"algo-1\",\"algo-2\"],\"network_interface_name\":\"eth0\"},\"user_entry_point\":\"train_tensorflow_smdataparallel_mnist.py\"}\u001b[0m\n",
      "\u001b[34mSM_USER_ARGS=[\"--model_dir\",\"s3://sagemaker-us-east-1-835319576252/tensorflow2-smdataparallel-mnist-2020-12-10-20-14-08-681/model\"]\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate\u001b[0m\n",
      "\u001b[34mSM_HP_MODEL_DIR=s3://sagemaker-us-east-1-835319576252/tensorflow2-smdataparallel-mnist-2020-12-10-20-14-08-681/model\u001b[0m\n",
      "\u001b[34mPYTHONPATH=/opt/ml/code:/usr/local/bin:/usr/local/lib/python37.zip:/usr/local/lib/python3.7:/usr/local/lib/python3.7/lib-dynload:/usr/local/lib/python3.7/site-packages\n",
      "\u001b[0m\n",
      "\u001b[34mInvoking script with the following command:\n",
      "\u001b[0m\n",
      "\u001b[34mmpirun --host algo-1:8,algo-2:8 -np 16 --allow-run-as-root --tag-output --oversubscribe -mca btl_tcp_if_include eth0 -mca oob_tcp_if_include eth0 -mca plm_rsh_no_tree_spawn 1 -mca pml ob1 -mca btl ^openib -mca orte_abort_on_non_zero_status 1 -mca plm_rsh_num_concurrent 2 -x NCCL_SOCKET_IFNAME=eth0 -x LD_LIBRARY_PATH -x PATH -x SMDATAPARALLEL_USE_HOMOGENEOUS=1 -x FI_PROVIDER=sockets -x RDMAV_FORK_SAFE=1 -x LD_PRELOAD=/usr/local/lib/python3.7/site-packages/gethostname.cpython-37m-x86_64-linux-gnu.so -x SMDATAPARALLEL_SERVER_ADDR=algo-1 -x SMDATAPARALLEL_SERVER_PORT=7592 -x SAGEMAKER_INSTANCE_TYPE=ml.p3.16xlarge smddprun /usr/local/bin/python3.7 -m mpi4py train_tensorflow_smdataparallel_mnist.py --model_dir s3://sagemaker-us-east-1-835319576252/tensorflow2-smdataparallel-mnist-2020-12-10-20-14-08-681/model\n",
      "\n",
      "\u001b[0m\n",
      "\u001b[35m2020-12-10 20:19:54,139 sagemaker-training-toolkit INFO     Process[es]: [psutil.Process(pid=170, name='orted', status='disk-sleep', started='20:19:53')]\u001b[0m\n",
      "\u001b[35m2020-12-10 20:19:54,139 sagemaker-training-toolkit INFO     Orted process found [psutil.Process(pid=170, name='orted', status='disk-sleep', started='20:19:53')]\u001b[0m\n",
      "\u001b[35m2020-12-10 20:19:54,139 sagemaker-training-toolkit INFO     Waiting for orted process [psutil.Process(pid=170, name='orted', status='disk-sleep', started='20:19:53')]\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "2020-12-10 20:19:56 Training - Training image download completed. Training in progress.\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Bootstrap : Using [0]eth0:10.2.234.77<0>\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Bootstrap : Using [0]eth0:10.2.228.56<0>\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO NET/Socket : Using [0]eth0:10.2.234.77<0>\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO NET/Socket : Using [0]eth0:10.2.228.56<0>\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:NCCL version 2.7.8+cuda11.0\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:NCCL version 2.7.8+cuda11.0\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Bootstrap : Using [0]eth0:10.2.234.77<0>\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Bootstrap : Using [0]eth0:10.2.234.77<0>\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Bootstrap : Using [0]eth0:10.2.234.77<0>\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO NET/Socket : Using [0]eth0:10.2.234.77<0>\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO NET/Socket : Using [0]eth0:10.2.234.77<0>\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO NET/Socket : Using [0]eth0:10.2.234.77<0>\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Bootstrap : Using [0]eth0:10.2.228.56<0>\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Bootstrap : Using [0]eth0:10.2.228.56<0>\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Bootstrap : Using [0]eth0:10.2.228.56<0>\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO NET/Socket : Using [0]eth0:10.2.228.56<0>\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO NET/Socket : Using [0]eth0:10.2.228.56<0>\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO NET/Socket : Using [0]eth0:10.2.228.56<0>\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Bootstrap : Using [0]eth0:10.2.234.77<0>\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Bootstrap : Using [0]eth0:10.2.234.77<0>\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Bootstrap : Using [0]eth0:10.2.234.77<0>\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Bootstrap : Using [0]eth0:10.2.234.77<0>\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO NET/Socket : Using [0]eth0:10.2.234.77<0>\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO NET/Socket : Using [0]eth0:10.2.234.77<0>\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO NET/Socket : Using [0]eth0:10.2.234.77<0>\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO NET/Socket : Using [0]eth0:10.2.234.77<0>\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Bootstrap : Using [0]eth0:10.2.228.56<0>\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Bootstrap : Using [0]eth0:10.2.228.56<0>\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Bootstrap : Using [0]eth0:10.2.228.56<0>\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Bootstrap : Using [0]eth0:10.2.228.56<0>\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO NET/Socket : Using [0]eth0:10.2.228.56<0>\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO NET/Socket : Using [0]eth0:10.2.228.56<0>\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO NET/Socket : Using [0]eth0:10.2.228.56<0>\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO NET/Socket : Using [0]eth0:10.2.228.56<0>\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Trees [0] 1/-1/-1->2->3|3->2->1/-1/-1 [1] 1/-1/-1->2->3|3->2->1/-1/-1 [2] 3/-1/-1->2->1|1->2->3/-1/-1 [3] 3/-1/-1->2->1|1->2->3/-1/-1 [4] -1/-1/-1->2->6|6->2->-1/-1/-1 [5] 6/-1/-1->2->0|0->2->6/-1/-1 [6] 1/-1/-1->2->3|3->2->1/-1/-1 [7] 1/-1/-1->2->3|3->2->1/-1/-1 [8] 3/-1/-1->2->1|1->2->3/-1/-1 [9] 3/-1/-1->2->1|1->2->3/-1/-1 [10] -1/-1/-1->2->6|6->2->-1/-1/-1 [11] 6/-1/-1->2->0|0->2->6/-1/-1\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 00/12 :    0   3   2   1   5   6   7   4\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 01/12 :    0   3   2   1   5   6   7   4\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 02/12 :    0   4   7   6   5   1   2   3\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 03/12 :    0   4   7   6   5   1   2   3\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 04/12 :    0   1   3   7   5   4   6   2\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 05/12 :    0   2   6   4   5   7   3   1\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 06/12 :    0   3   2   1   5   6   7   4\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 07/12 :    0   3   2   1   5   6   7   4\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 08/12 :    0   4   7   6   5   1   2   3\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 09/12 :    0   4   7   6   5   1   2   3\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 10/12 :    0   1   3   7   5   4   6   2\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Trees [0] 5/-1/-1->1->2|2->1->5/-1/-1 [1] 5/-1/-1->1->2|2->1->5/-1/-1 [2] 2/-1/-1->1->5|5->1->2/-1/-1 [3] 2/-1/-1->1->5|5->1->2/-1/-1 [4] 3/-1/-1->1->0|0->1->3/-1/-1 [5] -1/-1/-1->1->3|3->1->-1/-1/-1 [6] 5/-1/-1->1->2|2->1->5/-1/-1 [7] 5/-1/-1->1->2|2->1->5/-1/-1 [8] 2/-1/-1->1->5|5->1->2/-1/-1 [9] 2/-1/-1->1->5|5->1->2/-1/-1 [10] 3/-1/-1->1->0|0->1->3/-1/-1 [11] -1/-1/-1->1->3|3->1->-1/-1/-1\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Trees [0] 6/-1/-1->5->1|1->5->6/-1/-1 [1] 6/-1/-1->5->1|1->5->6/-1/-1 [2] 1/-1/-1->5->6|6->5->1/-1/-1 [3] 1/-1/-1->5->6|6->5->1/-1/-1 [4] 4/-1/-1->5->7|7->5->4/-1/-1 [5] 7/-1/-1->5->4|4->5->7/-1/-1 [6] 6/-1/-1->5->1|1->5->6/-1/-1 [7] 6/-1/-1->5->1|1->5->6/-1/-1 [8] 1/-1/-1->5->6|6->5->1/-1/-1 [9] 1/-1/-1->5->6|6->5->1/-1/-1 [10] 4/-1/-1->5->7|7->5->4/-1/-1 [11] 7/-1/-1->5->4|4->5->7/-1/-1\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 11/12 :    0   2   6   4   5   7   3   1\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Trees [0] 2/-1/-1->3->0|0->3->2/-1/-1 [1] 2/-1/-1->3->0|0->3->2/-1/-1 [2] -1/-1/-1->3->2|2->3->-1/-1/-1 [3] -1/-1/-1->3->2|2->3->-1/-1/-1 [4] 7/-1/-1->3->1|1->3->7/-1/-1 [5] 1/-1/-1->3->7|7->3->1/-1/-1 [6] 2/-1/-1->3->0|0->3->2/-1/-1 [7] 2/-1/-1->3->0|0->3->2/-1/-1 [8] -1/-1/-1->3->2|2->3->-1/-1/-1 [9] -1/-1/-1->3->2|2->3->-1/-1/-1 [10] 7/-1/-1->3->1|1->3->7/-1/-1 [11] 1/-1/-1->3->7|7->3->1/-1/-1\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Trees [0] -1/-1/-1->4->7|7->4->-1/-1/-1 [1] -1/-1/-1->4->7|7->4->-1/-1/-1 [2] 7/-1/-1->4->0|0->4->7/-1/-1 [3] 7/-1/-1->4->0|0->4->7/-1/-1 [4] 6/-1/-1->4->5|5->4->6/-1/-1 [5] 5/-1/-1->4->6|6->4->5/-1/-1 [6] -1/-1/-1->4->7|7->4->-1/-1/-1 [7] -1/-1/-1->4->7|7->4->-1/-1/-1 [8] 7/-1/-1->4->0|0->4->7/-1/-1 [9] 7/-1/-1->4->0|0->4->7/-1/-1 [10] 6/-1/-1->4->5|5->4->6/-1/-1 [11] 5/-1/-1->4->6|6->4->5/-1/-1\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Trees [0] 3/-1/-1->0->-1|-1->0->3/-1/-1 [1] 3/-1/-1->0->-1|-1->0->3/-1/-1 [2] 4/-1/-1->0->-1|-1->0->4/-1/-1 [3] 4/-1/-1->0->-1|-1->0->4/-1/-1 [4] 1/-1/-1->0->-1|-1->0->1/-1/-1 [5] 2/-1/-1->0->-1|-1->0->2/-1/-1 [6] 3/-1/-1->0->-1|-1->0->3/-1/-1 [7] 3/-1/-1->0->-1|-1->0->3/-1/-1 [8] 4/-1/-1->0->-1|-1->0->4/-1/-1 [9] 4/-1/-1->0->-1|-1->0->4/-1/-1 [10] 1/-1/-1->0->-1|-1->0->1/-1/-1 [11] 2/-1/-1->0->-1|-1->0->2/-1/-1\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Trees [0] 4/-1/-1->7->6|6->7->4/-1/-1 [1] 4/-1/-1->7->6|6->7->4/-1/-1 [2] 6/-1/-1->7->4|4->7->6/-1/-1 [3] 6/-1/-1->7->4|4->7->6/-1/-1 [4] 5/-1/-1->7->3|3->7->5/-1/-1 [5] 3/-1/-1->7->5|5->7->3/-1/-1 [6] 4/-1/-1->7->6|6->7->4/-1/-1 [7] 4/-1/-1->7->6|6->7->4/-1/-1 [8] 6/-1/-1->7->4|4->7->6/-1/-1 [9] 6/-1/-1->7->4|4->7->6/-1/-1 [10] 5/-1/-1->7->3|3->7->5/-1/-1 [11] 3/-1/-1->7->5|5->7->3/-1/-1\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Trees [0] 7/-1/-1->6->5|5->6->7/-1/-1 [1] 7/-1/-1->6->5|5->6->7/-1/-1 [2] 5/-1/-1->6->7|7->6->5/-1/-1 [3] 5/-1/-1->6->7|7->6->5/-1/-1 [4] 2/-1/-1->6->4|4->6->2/-1/-1 [5] 4/-1/-1->6->2|2->6->4/-1/-1 [6] 7/-1/-1->6->5|5->6->7/-1/-1 [7] 7/-1/-1->6->5|5->6->7/-1/-1 [8] 5/-1/-1->6->7|7->6->5/-1/-1 [9] 5/-1/-1->6->7|7->6->5/-1/-1 [10] 2/-1/-1->6->4|4->6->2/-1/-1 [11] 4/-1/-1->6->2|2->6->4/-1/-1\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 00 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 00 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 00 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 00 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 00 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 00 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 00 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 00 : 0[170] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 00 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 00 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 00 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 00 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 00 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 00 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 00 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 01 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 00/12 :    0   3   2   1   5   6   7   4\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 01/12 :    0   3   2   1   5   6   7   4\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 02/12 :    0   4   7   6   5   1   2   3\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 03/12 :    0   4   7   6   5   1   2   3\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 04/12 :    0   1   3   7   5   4   6   2\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Trees [0] 5/-1/-1->1->2|2->1->5/-1/-1 [1] 5/-1/-1->1->2|2->1->5/-1/-1 [2] 2/-1/-1->1->5|5->1->2/-1/-1 [3] 2/-1/-1->1->5|5->1->2/-1/-1 [4] 3/-1/-1->1->0|0->1->3/-1/-1 [5] -1/-1/-1->1->3|3->1->-1/-1/-1 [6] 5/-1/-1->1->2|2->1->5/-1/-1 [7] 5/-1/-1->1->2|2->1->5/-1/-1 [8] 2/-1/-1->1->5|5->1->2/-1/-1 [9] 2/-1/-1->1->5|5->1->2/-1/-1 [10] 3/-1/-1->1->0|0->1->3/-1/-1 [11] -1/-1/-1->1->3|3->1->-1/-1/-1\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 05/12 :    0   2   6   4   5   7   3   1\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 06/12 :    0   3   2   1   5   6   7   4\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Trees [0] 1/-1/-1->2->3|3->2->1/-1/-1 [1] 1/-1/-1->2->3|3->2->1/-1/-1 [2] 3/-1/-1->2->1|1->2->3/-1/-1 [3] 3/-1/-1->2->1|1->2->3/-1/-1 [4] -1/-1/-1->2->6|6->2->-1/-1/-1 [5] 6/-1/-1->2->0|0->2->6/-1/-1 [6] 1/-1/-1->2->3|3->2->1/-1/-1 [7] 1/-1/-1->2->3|3->2->1/-1/-1 [8] 3/-1/-1->2->1|1->2->3/-1/-1 [9] 3/-1/-1->2->1|1->2->3/-1/-1 [10] -1/-1/-1->2->6|6->2->-1/-1/-1 [11] 6/-1/-1->2->0|0->2->6/-1/-1\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 07/12 :    0   3   2   1   5   6   7   4\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 08/12 :    0   4   7   6   5   1   2   3\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 09/12 :    0   4   7   6   5   1   2   3\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Trees [0] 2/-1/-1->3->0|0->3->2/-1/-1 [1] 2/-1/-1->3->0|0->3->2/-1/-1 [2] -1/-1/-1->3->2|2->3->-1/-1/-1 [3] -1/-1/-1->3->2|2->3->-1/-1/-1 [4] 7/-1/-1->3->1|1->3->7/-1/-1 [5] 1/-1/-1->3->7|7->3->1/-1/-1 [6] 2/-1/-1->3->0|0->3->2/-1/-1 [7] 2/-1/-1->3->0|0->3->2/-1/-1 [8] -1/-1/-1->3->2|2->3->-1/-1/-1 [9] -1/-1/-1->3->2|2->3->-1/-1/-1 [10] 7/-1/-1->3->1|1->3->7/-1/-1 [11] 1/-1/-1->3->7|7->3->1/-1/-1\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Trees [0] -1/-1/-1->4->7|7->4->-1/-1/-1 [1] -1/-1/-1->4->7|7->4->-1/-1/-1 [2] 7/-1/-1->4->0|0->4->7/-1/-1 [3] 7/-1/-1->4->0|0->4->7/-1/-1 [4] 6/-1/-1->4->5|5->4->6/-1/-1 [5] 5/-1/-1->4->6|6->4->5/-1/-1 [6] -1/-1/-1->4->7|7->4->-1/-1/-1 [7] -1/-1/-1->4->7|7->4->-1/-1/-1 [8] 7/-1/-1->4->0|0->4->7/-1/-1 [9] 7/-1/-1->4->0|0->4->7/-1/-1 [10] 6/-1/-1->4->5|5->4->6/-1/-1 [11] 5/-1/-1->4->6|6->4->5/-1/-1\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 10/12 :    0   1   3   7   5   4   6   2\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 11/12 :    0   2   6   4   5   7   3   1\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Trees [0] 6/-1/-1->5->1|1->5->6/-1/-1 [1] 6/-1/-1->5->1|1->5->6/-1/-1 [2] 1/-1/-1->5->6|6->5->1/-1/-1 [3] 1/-1/-1->5->6|6->5->1/-1/-1 [4] 4/-1/-1->5->7|7->5->4/-1/-1 [5] 7/-1/-1->5->4|4->5->7/-1/-1 [6] 6/-1/-1->5->1|1->5->6/-1/-1 [7] 6/-1/-1->5->1|1->5->6/-1/-1 [8] 1/-1/-1->5->6|6->5->1/-1/-1 [9] 1/-1/-1->5->6|6->5->1/-1/-1 [10] 4/-1/-1->5->7|7->5->4/-1/-1 [11] 7/-1/-1->5->4|4->5->7/-1/-1\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Trees [0] 4/-1/-1->7->6|6->7->4/-1/-1 [1] 4/-1/-1->7->6|6->7->4/-1/-1 [2] 6/-1/-1->7->4|4->7->6/-1/-1 [3] 6/-1/-1->7->4|4->7->6/-1/-1 [4] 5/-1/-1->7->3|3->7->5/-1/-1 [5] 3/-1/-1->7->5|5->7->3/-1/-1 [6] 4/-1/-1->7->6|6->7->4/-1/-1 [7] 4/-1/-1->7->6|6->7->4/-1/-1 [8] 6/-1/-1->7->4|4->7->6/-1/-1 [9] 6/-1/-1->7->4|4->7->6/-1/-1 [10] 5/-1/-1->7->3|3->7->5/-1/-1 [11] 3/-1/-1->7->5|5->7->3/-1/-1\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Trees [0] 7/-1/-1->6->5|5->6->7/-1/-1 [1] 7/-1/-1->6->5|5->6->7/-1/-1 [2] 5/-1/-1->6->7|7->6->5/-1/-1 [3] 5/-1/-1->6->7|7->6->5/-1/-1 [4] 2/-1/-1->6->4|4->6->2/-1/-1 [5] 4/-1/-1->6->2|2->6->4/-1/-1 [6] 7/-1/-1->6->5|5->6->7/-1/-1 [7] 7/-1/-1->6->5|5->6->7/-1/-1 [8] 5/-1/-1->6->7|7->6->5/-1/-1 [9] 5/-1/-1->6->7|7->6->5/-1/-1 [10] 2/-1/-1->6->4|4->6->2/-1/-1 [11] 4/-1/-1->6->2|2->6->4/-1/-1\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Trees [0] 3/-1/-1->0->-1|-1->0->3/-1/-1 [1] 3/-1/-1->0->-1|-1->0->3/-1/-1 [2] 4/-1/-1->0->-1|-1->0->4/-1/-1 [3] 4/-1/-1->0->-1|-1->0->4/-1/-1 [4] 1/-1/-1->0->-1|-1->0->1/-1/-1 [5] 2/-1/-1->0->-1|-1->0->2/-1/-1 [6] 3/-1/-1->0->-1|-1->0->3/-1/-1 [7] 3/-1/-1->0->-1|-1->0->3/-1/-1 [8] 4/-1/-1->0->-1|-1->0->4/-1/-1 [9] 4/-1/-1->0->-1|-1->0->4/-1/-1 [10] 1/-1/-1->0->-1|-1->0->1/-1/-1 [11] 2/-1/-1->0->-1|-1->0->2/-1/-1\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 01 : 0[170] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 01 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 01 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 01 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 01 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 01 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 01 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 00 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 00 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 00 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 00 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 00 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 00 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 00 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 00 : 0[170] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 01 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 01 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 01 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 01 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 01 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 01 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 01 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 00 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 02 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 00 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 00 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 00 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 02 : 0[170] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 00 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 00 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 00 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 02 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 02 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 02 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 02 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 02 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 02 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 01 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 01 : 0[170] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 02 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 01 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 01 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 01 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 02 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 01 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 01 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 01 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 02 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 02 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 02 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 02 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 02 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 01 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 03 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 03 : 0[170] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 01 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 01 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 01 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 01 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 01 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 01 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 03 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 03 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 03 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 03 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 03 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 03 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 02 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 03 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 02 : 0[170] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 02 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 02 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 02 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 02 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 02 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 02 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 03 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 03 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 03 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 03 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 03 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 03 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 04 : 3[1a0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 02 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 02 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 04 : 0[170] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 02 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 02 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 02 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 04 : 4[1b0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 04 : 2[190] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 04 : 1[180] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 02 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 02 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 04 : 6[1d0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 04 : 5[1c0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 04 : 7[1e0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 03 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 03 : 0[170] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 03 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 03 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 03 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 03 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 04 : 2[190] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 03 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 03 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 04 : 3[1a0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 04 : 4[1b0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 04 : 1[180] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 04 : 6[1d0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 04 : 5[1c0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 04 : 7[1e0] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 03 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 03 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 03 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 05 : 2[190] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 03 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 03 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 03 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 05 : 0[170] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 03 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 05 : 3[1a0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 05 : 4[1b0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 05 : 1[180] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 05 : 5[1c0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 05 : 6[1d0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 05 : 7[1e0] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 04 : 3[1a0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 04 : 0[170] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 04 : 4[1b0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 04 : 1[180] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 04 : 2[190] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 04 : 5[1c0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 05 : 1[180] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 05 : 2[190] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 04 : 7[1e0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 04 : 6[1d0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 05 : 3[1a0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 05 : 4[1b0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 05 : 5[1c0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 05 : 6[1d0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 05 : 7[1e0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 06 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 04 : 2[190] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 06 : 0[170] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 04 : 3[1a0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 04 : 4[1b0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 04 : 1[180] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 06 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 04 : 5[1c0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 06 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 04 : 7[1e0] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 04 : 6[1d0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 06 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 06 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 06 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 06 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 05 : 2[190] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 05 : 0[170] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 06 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 06 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 05 : 4[1b0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 05 : 3[1a0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 05 : 1[180] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 06 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 05 : 5[1c0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 06 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 06 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 06 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 05 : 7[1e0] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 05 : 6[1d0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 06 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 05 : 1[180] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 07 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 05 : 2[190] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 07 : 0[170] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 07 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 07 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 05 : 3[1a0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 05 : 4[1b0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 07 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 05 : 5[1c0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 07 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 07 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 05 : 7[1e0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 05 : 6[1d0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 07 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 06 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 06 : 0[170] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 07 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 07 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 07 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 06 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 07 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 06 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 06 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 06 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 07 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 07 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 07 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 06 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 06 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 08 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 06 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 08 : 0[170] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 06 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 08 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 08 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 06 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 06 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 08 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 06 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 08 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 08 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 06 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 06 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 08 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 07 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 08 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 07 : 0[170] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 08 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 08 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 07 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 08 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 07 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 07 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 07 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 08 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 08 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 07 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 07 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 08 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 09 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 07 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 09 : 0[170] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 07 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 07 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 09 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 09 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 07 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 09 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 07 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 09 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 07 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 07 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 09 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 09 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 09 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 08 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 08 : 0[170] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 08 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 08 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 09 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 09 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 09 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 08 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 08 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 09 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 09 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 08 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 08 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 09 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 10 : 3[1a0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 08 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 10 : 0[170] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 08 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 08 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 10 : 1[180] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 10 : 2[190] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 08 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 10 : 4[1b0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 10 : 5[1c0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 08 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 10 : 6[1d0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 08 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 08 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 10 : 7[1e0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 09 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 10 : 2[190] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 09 : 0[170] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 10 : 3[1a0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 10 : 1[180] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 09 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 09 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 10 : 4[1b0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 09 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 10 : 5[1c0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 10 : 6[1d0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 09 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 10 : 7[1e0] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 09 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 09 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 09 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 11 : 2[190] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 11 : 0[170] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 09 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 11 : 1[180] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 09 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 09 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 11 : 4[1b0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 11 : 3[1a0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 09 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 11 : 5[1c0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 11 : 6[1d0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 09 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 09 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 11 : 7[1e0] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 10 : 3[1a0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 11 : 1[180] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 10 : 0[170] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 11 : 2[190] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 10 : 1[180] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 10 : 4[1b0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 10 : 2[190] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 11 : 4[1b0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 11 : 3[1a0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 10 : 5[1c0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 11 : 5[1c0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 11 : 6[1d0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 10 : 6[1d0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 10 : 7[1e0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 11 : 7[1e0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO comm 0x556adae8ba80 rank 1 nranks 8 cudaDev 1 busId 180 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO comm 0x5619f7e04430 rank 0 nranks 8 cudaDev 0 busId 170 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO comm 0x560c7c854450 rank 2 nranks 8 cudaDev 2 busId 190 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO comm 0x55c7843bf570 rank 3 nranks 8 cudaDev 3 busId 1a0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO comm 0x5612c64e3c70 rank 4 nranks 8 cudaDev 4 busId 1b0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO comm 0x55eebfe4bed0 rank 5 nranks 8 cudaDev 5 busId 1c0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO comm 0x561623d8a800 rank 6 nranks 8 cudaDev 6 busId 1d0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO comm 0x557a3be80650 rank 7 nranks 8 cudaDev 7 busId 1e0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 10 : 2[190] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 10 : 1[180] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 10 : 3[1a0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 10 : 4[1b0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 10 : 5[1c0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 10 : 7[1e0] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 10 : 6[1d0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 11 : 2[190] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 11 : 0[170] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 11 : 1[180] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 11 : 3[1a0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 11 : 4[1b0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 11 : 5[1c0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 11 : 6[1d0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 11 : 7[1e0] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 11 : 1[180] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 11 : 2[190] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 11 : 3[1a0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 11 : 4[1b0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 11 : 5[1c0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 11 : 6[1d0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 11 : 7[1e0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO comm 0x5555f70aa450 rank 1 nranks 8 cudaDev 1 busId 180 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO comm 0x557be7278a90 rank 0 nranks 8 cudaDev 0 busId 170 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO comm 0x55f9e68b7a20 rank 2 nranks 8 cudaDev 2 busId 190 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO comm 0x5562e799fb70 rank 3 nranks 8 cudaDev 3 busId 1a0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO comm 0x562e518cb400 rank 4 nranks 8 cudaDev 4 busId 1b0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO comm 0x564cac31d2b0 rank 5 nranks 8 cudaDev 5 busId 1c0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO comm 0x55a633789ca0 rank 6 nranks 8 cudaDev 6 busId 1d0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO comm 0x555a3b520360 rank 7 nranks 8 cudaDev 7 busId 1e0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Trees [0] 2/8/-1->3->0|0->3->2/8/-1 [1] 2/-1/-1->3->0|0->3->2/-1/-1\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Trees [0] -1/-1/-1->4->7|7->4->-1/-1/-1 [1] -1/-1/-1->4->7|7->4->-1/-1/-1\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Trees [0] 6/-1/-1->5->1|1->5->6/-1/-1 [1] 6/-1/-1->5->1|1->5->6/-1/-1\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Trees [0] 7/-1/-1->6->5|5->6->7/-1/-1 [1] 7/-1/-1->6->5|5->6->7/-1/-1\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Trees [0] 4/-1/-1->7->6|6->7->4/-1/-1 [1] 4/-1/-1->7->6|6->7->4/-1/-1\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 00/02 :    0   3   2   1   5   6   7   4   8  11  10   9  13  14  15  12\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 01/02 :    0   3   2   1   5   6   7   4   8  11  10   9  13  14  15  12\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Trees [0] 5/-1/-1->1->2|2->1->5/-1/-1 [1] 5/-1/-1->1->2|2->1->5/-1/-1\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Trees [0] 1/-1/-1->2->3|3->2->1/-1/-1 [1] 1/-1/-1->2->3|3->2->1/-1/-1\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Trees [0] -1/-1/-1->12->15|15->12->-1/-1/-1 [1] -1/-1/-1->12->15|15->12->-1/-1/-1\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Trees [0] 3/-1/-1->0->-1|-1->0->3/-1/-1 [1] 3/-1/-1->0->11|11->0->3/-1/-1\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Trees [0] 15/-1/-1->14->13|13->14->15/-1/-1 [1] 15/-1/-1->14->13|13->14->15/-1/-1\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Trees [0] 14/-1/-1->13->9|9->13->14/-1/-1 [1] 14/-1/-1->13->9|9->13->14/-1/-1\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Trees [0] 10/-1/-1->11->8|8->11->10/-1/-1 [1] 10/0/-1->11->8|8->11->10/0/-1\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Trees [0] 9/-1/-1->10->11|11->10->9/-1/-1 [1] 9/-1/-1->10->11|11->10->9/-1/-1\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Trees [0] 13/-1/-1->9->10|10->9->13/-1/-1 [1] 13/-1/-1->9->10|10->9->13/-1/-1\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Trees [0] 11/-1/-1->8->3|3->8->11/-1/-1 [1] 11/-1/-1->8->-1|-1->8->11/-1/-1\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/64\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Trees [0] 12/-1/-1->15->14|14->15->12/-1/-1 [1] 12/-1/-1->15->14|14->15->12/-1/-1\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 00 : 9[180] -> 13[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 00 : 13[1c0] -> 14[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 00 : 11[1a0] -> 10[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 00 : 10[190] -> 9[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 00 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 00 : 14[1d0] -> 15[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 00 : 15[1e0] -> 12[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 00 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 00 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 00 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 00 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 00 : 4[1b0] -> 8[170] [receive] via NET/Socket/0\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 00 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 00 : 12[1b0] -> 0[170] [receive] via NET/Socket/0\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 00 : 12[1b0] -> 0[170] [send] via NET/Socket/0\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 00 : 4[1b0] -> 8[170] [send] via NET/Socket/0\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 00 : 8[170] -> 11[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 00 : 0[170] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 00 : 9[180] -> 10[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 00 : 13[1c0] -> 9[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 00 : 10[190] -> 11[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 00 : 14[1d0] -> 13[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 00 : 15[1e0] -> 14[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 00 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 00 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 00 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 00 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 00 : 12[1b0] -> 15[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 00 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 00 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 00 : 11[1a0] -> 8[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 00 : 8[170] -> 3[1a0] [send] via NET/Socket/0\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 00 : 8[170] -> 3[1a0] [receive] via NET/Socket/0\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 00 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 01 : 9[180] -> 13[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 01 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 01 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 01 : 13[1c0] -> 14[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 01 : 10[190] -> 9[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 01 : 14[1d0] -> 15[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 01 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 01 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 01 : 15[1e0] -> 12[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 01 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 01 : 11[1a0] -> 10[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 01 : 12[1b0] -> 0[170] [send] via NET/Socket/0\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 01 : 4[1b0] -> 8[170] [send] via NET/Socket/0\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 01 : 12[1b0] -> 0[170] [receive] via NET/Socket/0\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 00 : 3[1a0] -> 8[170] [receive] via NET/Socket/0\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO Channel 01 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO Channel 01 : 9[180] -> 10[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 00 : 3[1a0] -> 8[170] [send] via NET/Socket/0\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO Channel 01 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO Channel 01 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO Channel 01 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO Channel 01 : 13[1c0] -> 9[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO Channel 01 : 14[1d0] -> 13[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO Channel 01 : 10[190] -> 11[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO Channel 01 : 15[1e0] -> 14[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 01 : 0[170] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:algo-1:178:178 [6] NCCL INFO comm 0x55a63645ac90 rank 6 nranks 16 cudaDev 6 busId 1d0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:algo-1:177:177 [5] NCCL INFO comm 0x564caefee1a0 rank 5 nranks 16 cudaDev 5 busId 1c0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:algo-2:178:178 [1] NCCL INFO comm 0x556addb5ca70 rank 9 nranks 16 cudaDev 1 busId 180 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:algo-2:186:186 [5] NCCL INFO comm 0x55eec2b1cec0 rank 13 nranks 16 cudaDev 5 busId 1c0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:algo-2:187:187 [6] NCCL INFO comm 0x561626a5b7f0 rank 14 nranks 16 cudaDev 6 busId 1d0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 01 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO Channel 01 : 12[1b0] -> 15[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:algo-2:188:188 [7] NCCL INFO comm 0x557a3eb51640 rank 15 nranks 16 cudaDev 7 busId 1e0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:algo-2:184:184 [4] NCCL INFO comm 0x5612c91b4c60 rank 12 nranks 16 cudaDev 4 busId 1b0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO Channel 01 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO Channel 01 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 01 : 4[1b0] -> 8[170] [receive] via NET/Socket/0\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:algo-1:169:169 [1] NCCL INFO comm 0x5555f9d7b440 rank 1 nranks 16 cudaDev 1 busId 180 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO Channel 01 : 8[170] -> 11[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:algo-1:172:172 [2] NCCL INFO comm 0x55f9e9588a10 rank 2 nranks 16 cudaDev 2 busId 190 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:algo-1:173:173 [3] NCCL INFO comm 0x5562ea670b60 rank 3 nranks 16 cudaDev 3 busId 1a0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 01 : 0[170] -> 11[1a0] [send] via NET/Socket/0\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO Channel 01 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:algo-2:181:181 [2] NCCL INFO comm 0x560c7f525440 rank 10 nranks 16 cudaDev 2 busId 190 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:algo-1:176:176 [4] NCCL INFO comm 0x562e5459c3f0 rank 4 nranks 16 cudaDev 4 busId 1b0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:algo-1:179:179 [7] NCCL INFO comm 0x555a3e1f1250 rank 7 nranks 16 cudaDev 7 busId 1e0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 01 : 0[170] -> 11[1a0] [receive] via NET/Socket/0\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 01 : 11[1a0] -> 8[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:176 [0] NCCL INFO comm 0x5619faad5420 rank 8 nranks 16 cudaDev 0 busId 170 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO Channel 01 : 11[1a0] -> 0[170] [receive] via NET/Socket/0\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO Channel 01 : 11[1a0] -> 0[170] [send] via NET/Socket/0\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:167 [0] NCCL INFO comm 0x557be9f49a80 rank 0 nranks 16 cudaDev 0 busId 170 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:algo-2:182:182 [3] NCCL INFO comm 0x55c787090560 rank 11 nranks 16 cudaDev 3 busId 1a0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:Running smdistributed.dataparallel v1.0.0\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz[1,15]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz[1,1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:Downloading data from [1,5]<stdout>:https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz[1,2]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz[1,6]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:Downloading data from [1,3]<stdout>:https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz[1,4]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:#015    8192/11490434 [..............................] - ETA: 0s[1,10]<stdout>:#015    8192/11490434 [..............................] - ETA: 0s[1,14]<stdout>:#015    8192/11490434 [..............................] - ETA: 0s[1,11]<stdout>:#015    8192/11490434 [..............................] - ETA: 0s[1,8]<stdout>:#015    8192/11490434 [..............................] - ETA: 0s[1,12]<stdout>:#015[1,12]<stdout>:    8192/11490434 [..............................] - ETA: 0s[1,13]<stdout>:#015    8192/11490434 [..............................] - ETA: 0s[1,9]<stdout>:#015[1,9]<stdout>:    8192/11490434 [..............................] - ETA: 0s[1,6]<stdout>:#015    8192/11490434 [..............................] - ETA: 0s[1,7]<stdout>:#015[1,7]<stdout>:    8192/11490434 [..............................] - ETA: 0s[1,1]<stdout>:#015[1,1]<stdout>:    8192/11490434 [..............................] - ETA: 0s[1,5]<stdout>:#015[1,5]<stdout>:    8192/11490434 [..............................] - ETA: 0s[1,0]<stdout>:#015    8192/11490434 [..............................][1,0]<stdout>: - ETA: 0s[1,2]<stdout>:#015    8192/11490434 [..............................][1,2]<stdout>: - ETA: 0s[1,4]<stdout>:#015[1,4]<stdout>:    8192/11490434 [..............................] - ETA: 0s[1,3]<stdout>:#015[1,3]<stdout>:    8192/11490434 [..............................] - ETA: 0s[1,15]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015 3678208/11490434 [========>.....................][1,15]<stdout>: - ETA: 0s[1,10]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015 5431296/11490434 [=============>................] - ETA: 0s[1,14]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015 8536064/11490434 [=====================>........] - ETA: 0s[1,11]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015 7061504/11490434 [=================>............] - ETA: 0s[1,8]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015 8192000/11490434 [====================>.........] - ETA: 0s[1,12]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015[1,12]<stdout>:  925696/11490434 [=>............................] - ETA: 0s[1,13]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015 7684096/11490434 [===================>..........] - ETA: 0s[1,9]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015[1,9]<stdout>: 8224768/11490434 [====================>.........] - ETA: 0s[1,6]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015 4145152/11490434 [=========>....................][1,6]<stdout>: - ETA: 0s[1,7]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015[1,7]<stdout>: 2818048/11490434 [======>.......................] - ETA: 0s[1,1]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015[1,1]<stdout>: 8224768/11490434 [====================>.........] - ETA: 0s[1,14]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01511493376/11490434 [==============================] - 0s 0us/step\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015[1,5]<stdout>: 2605056/11490434 [=====>........................] - ETA: 0s[1,0]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015[1,0]<stdout>: 1843200/11490434 [===>..........................] - ETA: 0s[1,2]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015[1,2]<stdout>: 3588096/11490434 [========>.....................] - ETA: 0s[1,8]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01511493376/11490434 [==============================] - 0s 0us/step\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015 7970816/11490434 [===================>..........][1,4]<stdout>: - ETA: 0s[1,11]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01511493376/11490434 [==============================] - 0s 0us/step\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01511493376/11490434 [==============================] - 0s 0us/step\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015[1,3]<stdout>: 8093696/11490434 [====================>.........] - ETA: 0s[1,1]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01511493376/11490434 [==============================] - 0s 0us/step\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01511493376/11490434 [==============================] - 0s 0us/step\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01511493376/11490434 [==============================] - 0s 0us/step\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01511493376/11490434 [==============================] - 0s 0us/step\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01510018816/11490434 [=========================>....] - ETA: 0s[1,3]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01511493376/11490434 [==============================][1,3]<stdout>: - 0s 0us/step\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015[1,12]<stdout>: 3325952/11490434 [=======>......................][1,12]<stdout>: - ETA: 0s[1,15]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01511493376/11490434 [==============================] - 0s 0us/step\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015[1,6]<stdout>: 9469952/11490434 [=======================>......] - ETA: 0s[1,7]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015[1,7]<stdout>: 6709248/11490434 [================>.............] - ETA: 0s[1,5]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015[1,5]<stdout>: 6168576/11490434 [===============>..............] - ETA: 0s[1,0]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015[1,0]<stdout>: 5144576/11490434 [============>.................] - ETA: 0s[1,2]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015[1,2]<stdout>: 9388032/11490434 [=======================>......] - ETA: 0s[1,6]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01511493376/11490434 [==============================] - 0s 0us/step\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01511493376/11490434 [==============================][1,2]<stdout>: - 0s 0us/step\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015[1,7]<stdout>:11493376/11490434 [==============================] - 0s 0us/step\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01511493376/11490434 [==============================][1,5]<stdout>: - 0s 0us/step\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01511198464/11490434 [============================>.] - ETA: 0s[1,12]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01511493376/11490434 [==============================] - 0s 0us/step\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015[1,0]<stdout>:10788864/11490434 [===========================>..] - ETA: 0s[1,0]<stdout>:#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01511493376/11490434 [==============================][1,0]<stdout>: - 0s 0us/step\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:[2020-12-10 20:20:04.904 algo-2:181 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:[2020-12-10 20:20:04.904 algo-2:186 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:[2020-12-10 20:20:04.904 algo-2:188 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:[2020-12-10 20:20:04.943 algo-1:169 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:[2020-12-10 20:20:04.943 algo-1:176 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:[2020-12-10 20:20:04.943 algo-1:179 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:[2020-12-10 20:20:04.943 algo-1:172 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:[2020-12-10 20:20:04.943 algo-1:173 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:[2020-12-10 20:20:04.943 algo-1:177 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:[2020-12-10 20:20:04.943 algo-1:178 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:[2020-12-10 20:20:04.943 algo-1:167 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:[2020-12-10 20:20:04.948 algo-2:184 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:[2020-12-10 20:20:04.951 algo-2:187 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:[2020-12-10 20:20:04.956 algo-2:182 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:[2020-12-10 20:20:04.963 algo-2:176 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:[2020-12-10 20:20:04.971 algo-2:178 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:[2020-12-10 20:20:05.200 algo-2:188 INFO profiler_config_parser.py:102] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:[2020-12-10 20:20:05.200 algo-2:187 INFO profiler_config_parser.py:102] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:[2020-12-10 20:20:05.200 algo-2:176 INFO profiler_config_parser.py:102] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:[2020-12-10 20:20:05.200 algo-2:184 INFO profiler_config_parser.py:102] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:[2020-12-10 20:20:05.200 algo-2:182 INFO profiler_config_parser.py:102] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:[2020-12-10 20:20:05.200 algo-2:186 INFO profiler_config_parser.py:102] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:[2020-12-10 20:20:05.200 algo-2:178 INFO profiler_config_parser.py:102] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:[2020-12-10 20:20:05.200 algo-2:181 INFO profiler_config_parser.py:102] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:[2020-12-10 20:20:05.201 algo-2:178 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:[2020-12-10 20:20:05.201 algo-2:188 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:[2020-12-10 20:20:05.201 algo-2:187 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:[2020-12-10 20:20:05.201 algo-2:176 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:[2020-12-10 20:20:05.201 algo-2:184 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:[2020-12-10 20:20:05.201 algo-2:182 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:[2020-12-10 20:20:05.201 algo-2:181 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:[2020-12-10 20:20:05.201 algo-2:186 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:[2020-12-10 20:20:05.202 algo-2:181 INFO hook.py:199] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:[2020-12-10 20:20:05.202 algo-2:182 INFO hook.py:199] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:[2020-12-10 20:20:05.202 algo-2:178 INFO hook.py:199] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:[2020-12-10 20:20:05.203 algo-2:188 INFO hook.py:199] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:[2020-12-10 20:20:05.203 algo-2:176 INFO hook.py:199] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:[2020-12-10 20:20:05.202 algo-2:184 INFO hook.py:199] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:[2020-12-10 20:20:05.203 algo-2:187 INFO hook.py:199] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:[2020-12-10 20:20:05.203 algo-2:186 INFO hook.py:199] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:[2020-12-10 20:20:05.258 algo-1:178 INFO profiler_config_parser.py:102] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:[2020-12-10 20:20:05.258 algo-1:169 INFO profiler_config_parser.py:102] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:[2020-12-10 20:20:05.258 algo-1:172 INFO profiler_config_parser.py:102] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:[2020-12-10 20:20:05.258 algo-1:173 INFO profiler_config_parser.py:102] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:[2020-12-10 20:20:05.258 algo-1:177 INFO profiler_config_parser.py:102] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:[2020-12-10 20:20:05.258 algo-1:167 INFO profiler_config_parser.py:102] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:[2020-12-10 20:20:05.258 algo-1:176 INFO profiler_config_parser.py:102] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:[2020-12-10 20:20:05.258 algo-1:179 INFO profiler_config_parser.py:102] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:[2020-12-10 20:20:05.260 algo-1:178 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:[2020-12-10 20:20:05.260 algo-1:167 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:[2020-12-10 20:20:05.260 algo-1:176 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:[2020-12-10 20:20:05.260 algo-1:169 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:[2020-12-10 20:20:05.260 algo-1:172 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:[2020-12-10 20:20:05.260 algo-1:173 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:[2020-12-10 20:20:05.260 algo-1:179 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:[2020-12-10 20:20:05.260 algo-1:177 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:[2020-12-10 20:20:05.260 algo-1:177 INFO hook.py:199] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:[2020-12-10 20:20:05.261 algo-1:178 INFO hook.py:199] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:[2020-12-10 20:20:05.261 algo-1:167 INFO hook.py:199] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:[2020-12-10 20:20:05.261 algo-1:176 INFO hook.py:199] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:[2020-12-10 20:20:05.261 algo-1:169 INFO hook.py:199] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:[2020-12-10 20:20:05.261 algo-1:172 INFO hook.py:199] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:[2020-12-10 20:20:05.261 algo-1:179 INFO hook.py:199] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:[2020-12-10 20:20:05.261 algo-1:173 INFO hook.py:199] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:[2020-12-10 20:20:05.300 algo-2:187 INFO hook.py:253] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:[2020-12-10 20:20:05.300 algo-2:181 INFO hook.py:253] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:[2020-12-10 20:20:05.300 algo-2:181 INFO state_store.py:67] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:[2020-12-10 20:20:05.300 algo-2:178 INFO hook.py:253] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:[2020-12-10 20:20:05.300 algo-2:178 INFO state_store.py:67] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:[2020-12-10 20:20:05.300 algo-2:184 INFO hook.py:253] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:[2020-12-10 20:20:05.300 algo-2:184 INFO state_store.py:67] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:[2020-12-10 20:20:05.300 algo-2:176 INFO hook.py:253] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:[2020-12-10 20:20:05.300 algo-2:176 INFO state_store.py:67] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:[2020-12-10 20:20:05.300 algo-2:186 INFO hook.py:253] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:[2020-12-10 20:20:05.300 algo-2:186 INFO state_store.py:67] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:[2020-12-10 20:20:05.300 algo-2:187 INFO state_store.py:67] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:[2020-12-10 20:20:05.300 algo-2:182 INFO hook.py:253] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:[2020-12-10 20:20:05.300 algo-2:182 INFO state_store.py:67] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:[2020-12-10 20:20:05.300 algo-2:188 INFO hook.py:253] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:[2020-12-10 20:20:05.300 algo-2:188 INFO state_store.py:67] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,10]<stdout>:[2020-12-10 20:20:05.301 algo-2:181 INFO hook.py:413] Monitoring the collections: metrics, sm_metrics, losses\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:[2020-12-10 20:20:05.301 algo-2:176 INFO hook.py:413] Monitoring the collections: sm_metrics, losses, metrics\u001b[0m\n",
      "\u001b[34m[1,14]<stdout>:[2020-12-10 20:20:05.301 algo-2:187 INFO hook.py:413] Monitoring the collections: metrics, sm_metrics, losses\u001b[0m\n",
      "\u001b[34m[1,9]<stdout>:[2020-12-10 20:20:05.301 algo-2:178 INFO hook.py:413] Monitoring the collections: metrics, sm_metrics, losses\u001b[0m\n",
      "\u001b[34m[1,12]<stdout>:[2020-12-10 20:20:05.301 algo-2:184 INFO hook.py:413] Monitoring the collections: sm_metrics, metrics, losses\u001b[0m\n",
      "\u001b[34m[1,13]<stdout>:[2020-12-10 20:20:05.301 algo-2:186 INFO hook.py:413] Monitoring the collections: losses, sm_metrics, metrics\u001b[0m\n",
      "\u001b[34m[1,11]<stdout>:[2020-12-10 20:20:05.301 algo-2:182 INFO hook.py:413] Monitoring the collections: metrics, sm_metrics, losses\u001b[0m\n",
      "\u001b[34m[1,15]<stdout>:[2020-12-10 20:20:05.301 algo-2:188 INFO hook.py:413] Monitoring the collections: metrics, sm_metrics, losses\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:[2020-12-10 20:20:05.348 algo-1:177 INFO hook.py:253] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:[2020-12-10 20:20:05.348 algo-1:178 INFO hook.py:253] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:[2020-12-10 20:20:05.348 algo-1:169 INFO hook.py:253] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:[2020-12-10 20:20:05.348 algo-1:173 INFO hook.py:253] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:[2020-12-10 20:20:05.348 algo-1:177 INFO state_store.py:67] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:[2020-12-10 20:20:05.348 algo-1:167 INFO hook.py:253] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:[2020-12-10 20:20:05.348 algo-1:176 INFO hook.py:253] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:[2020-12-10 20:20:05.348 algo-1:169 INFO state_store.py:67] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:[2020-12-10 20:20:05.348 algo-1:172 INFO hook.py:253] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:[2020-12-10 20:20:05.348 algo-1:176 INFO state_store.py:67] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:[2020-12-10 20:20:05.348 algo-1:167 INFO state_store.py:67] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:[2020-12-10 20:20:05.348 algo-1:178 INFO state_store.py:67] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:[2020-12-10 20:20:05.348 algo-1:173 INFO state_store.py:67] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:[2020-12-10 20:20:05.348 algo-1:179 INFO hook.py:253] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:[2020-12-10 20:20:05.349 algo-1:172 INFO state_store.py:67] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:[2020-12-10 20:20:05.349 algo-1:179 INFO state_store.py:67] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,5]<stdout>:[2020-12-10 20:20:05.349 algo-1:177 INFO hook.py:413] Monitoring the collections: sm_metrics, losses, metrics\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:[2020-12-10 20:20:05.349 algo-1:167 INFO hook.py:413] Monitoring the collections: metrics, losses, sm_metrics\u001b[0m\n",
      "\u001b[34m[1,1]<stdout>:[2020-12-10 20:20:05.349 algo-1:169 INFO hook.py:413] Monitoring the collections: losses, metrics, sm_metrics\u001b[0m\n",
      "\u001b[34m[1,2]<stdout>:[2020-12-10 20:20:05.349 algo-1:172 INFO hook.py:413] Monitoring the collections: metrics, losses, sm_metrics\u001b[0m\n",
      "\u001b[34m[1,3]<stdout>:[2020-12-10 20:20:05.349 algo-1:173 INFO hook.py:413] Monitoring the collections: sm_metrics, losses, metrics\u001b[0m\n",
      "\u001b[34m[1,7]<stdout>:[2020-12-10 20:20:05.349 algo-1:179 INFO hook.py:413] Monitoring the collections: metrics, losses, sm_metrics\u001b[0m\n",
      "\u001b[34m[1,6]<stdout>:[2020-12-10 20:20:05.349 algo-1:178 INFO hook.py:413] Monitoring the collections: sm_metrics, losses, metrics\u001b[0m\n",
      "\u001b[34m[1,4]<stdout>:[2020-12-10 20:20:05.349 algo-1:176 INFO hook.py:413] Monitoring the collections: sm_metrics, losses, metrics\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34m[1,0]<stdout>:algo-1:167:1400 [0] NCCL INFO Launch mode Parallel\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:Step #0#011Loss: 2.306592\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:algo-1:167:1392 [0] NCCL INFO Launch mode Parallel\u001b[0m\n",
      "\u001b[34m[1,8]<stdout>:algo-2:176:1389 [0] NCCL INFO Launch mode Parallel\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:Step #50#011Loss: 0.265049\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:Step #100#011Loss: 0.245126\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:Step #150#011Loss: 0.153079\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:Step #200#011Loss: 0.214459\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:Step #250#011Loss: 0.087619\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:Step #300#011Loss: 0.077063\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:Step #350#011Loss: 0.049630\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:Step #400#011Loss: 0.181920\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:Step #450#011Loss: 0.114983\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:Step #500#011Loss: 0.183502\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:Step #550#011Loss: 0.066536\u001b[0m\n",
      "\u001b[34m[1,0]<stdout>:Step #600#011Loss: 0.089528\u001b[0m\n",
      "\u001b[35m2020-12-10 20:20:27,057 sagemaker-training-toolkit INFO     Orted process exited\u001b[0m\n",
      "\u001b[34m2020-12-10 20:20:27,036 sagemaker-training-toolkit INFO     Reporting training SUCCESS\u001b[0m\n",
      "\u001b[35m2020-12-10 20:20:57,087 sagemaker-training-toolkit INFO     MPI process finished.\u001b[0m\n",
      "\u001b[35m2020-12-10 20:20:57,088 sagemaker_tensorflow_container.training WARNING  No model artifact is saved under path /opt/ml/model. Your training job will not save any model files to S3.\u001b[0m\n",
      "\u001b[35mFor details of how to construct your training script see:\u001b[0m\n",
      "\u001b[35mhttps://sagemaker.readthedocs.io/en/stable/using_tf.html#adapting-your-local-tensorflow-script\u001b[0m\n",
      "\u001b[35m2020-12-10 20:20:57,088 sagemaker-training-toolkit INFO     Reporting training SUCCESS\u001b[0m\n",
      "\n",
      "2020-12-10 20:21:08 Uploading - Uploading generated training model\n",
      "2020-12-10 20:21:08 Completed - Training job completed\n",
      "Training seconds: 420\n",
      "Billable seconds: 420\n"
     ]
    }
   ],
   "source": [
    "estimator.fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "Now that you have a trained model, you can deploy an endpoint to host the model. After you deploy the endpoint, you can then test it with inference requests. The following cell will store the model_data variable to be used with the inference notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Storing s3://sagemaker-us-east-1-835319576252/tensorflow2-smdataparallel-mnist-2020-12-10-20-14-08-681/output/model.tar.gz as model_data\n",
      "Stored 'model_data' (str)\n"
     ]
    }
   ],
   "source": [
    "model_data = estimator.model_data\n",
    "print(\"Storing {} as model_data\".format(model_data))\n",
    "%store model_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_tensorflow_p36",
   "language": "python",
   "name": "conda_tensorflow_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
